Introduction

This live session has been adapted from from Edgar Ruiz’s A Gentle Introduction to tidymodels. We will run through an end-to-end modeling examples. The first will be a regression example using the Sacramento housing prices dataset, and the second will be a classification example using the GermanCredit credit score dataset.

I will briefly touch on important steps of the predictive modeling process, but this is not mean to be a comprehensive instruction on predictive modeling (a.k.a. machine learning). This tutorial is rather just meant to give a flavor of what’s possible using the new tidymodels universe of packages. For a much more detailed overview, see Max Kuhn’s fantastic Applied Predictive Modeling.

Install necessary pacakges

# pacman will help us install any necessary packages
if (!require("pacman")) install.packages("pacman")
Loading required package: pacman
# pacman::p_load checks to see if this packages are installed, and installs them if not
pacman::p_load(tidymodels, ranger, randomForest)

Load packages

Load the tidymodels library. This loads a collection of both tidymodels packages, and select tidyverse packages like dplyr, purrr, ggplot2. We will also load caret since it has some very nice datasets for regression and classification exercises.

library(tidymodels)
library(caret)
Loading required package: lattice
Registered S3 method overwritten by 'data.table':
  method           from
  print.data.table     

Attaching package: ‘caret’

The following objects are masked from ‘package:yardstick’:

    precision, recall

The following object is masked from ‘package:purrr’:

    lift
# Set random number seed to get consistent results
set.seed(101)

Regression example using Sacramento

Load dataset

data(Sacramento)

Sacramento
Sacramento

Data sampling

The first step in the modeling process is to split your data into separate training and testing datasets. The model will be trained using the training dataset, and the testing dataset will not be used until you are ready to assess model performance. The rsample::initial_split function helps with this initial splitting of the data. The rsample includes many other helpful functions for splitting the data for cross-validation, bootstrapping, etc.

# Split the dataset, using 75% of the data for training and 25% for testing
housing_split <- Sacramento %>%
  as_tibble() %>%
  dplyr::select(price,type,sqft,beds,baths,latitude,longitude) %>%
  initial_split(prop = 0.75)

# This rsplit object tells you how many observations are used for training, how many for testing, and how many total
housing_split
<699/233/932>
# The training function can be used to extract the training data from the rsplit object
housing_training <- housing_split %>%
  training()

housing_training

# The testing function can be used to extract the testing data from the rsplit object
housing_testing <- housing_split %>%
  testing()

housing_testing

Data pre-processing

After splitting the data, we will do some data processing. To do this, we will use the recipe package. A recipe is a blueprint for how data will be processed. By creating a blueprint, rather than processing data directly, we can apply the same blueprint to training and testing datasets. Importantly, the recipe is defined using only data from the training dataset, which will allow us to see how well the model performs using the testing dataset. Recipe steps can be defined using pipes with a number of sequential steps - there are many many options for recipe steps.

Model training

Next, we will use the parsnip package to define a number of models. Generally, we use parsnip to define 3 things about our model:

  1. The type of model (e.g., linear regression or random forest)
  2. the mode of the model (e.g., regression or classification)
  3. The engine for the model (e.g., ranger or randomForest)

After we’ve defined the model in this way, we can use the fit function to fit the model.

# Define and fit a linear regression model
housing_model_lm <- linear_reg() %>%
  set_engine("lm") 

housing_model_lm
Linear Regression Model Specification (regression)

Computational engine: lm 
housing_fit_lm <- housing_model_lm %>%
  fit(price ~ ., data = housing_training_juiced)

housing_fit_lm
parsnip model object


Call:
stats::lm(formula = formula, data = data)

Coefficients:
      (Intercept)               sqft               beds              baths  
           248531             106863             -22235               6528  
         latitude          longitude  type_Multi_Family   type_Residential  
             5953              17524              -4297              10493  
# Define and fit a random forest regression model using the randomForest engine/package
housing_model_randomForest <-  rand_forest(trees = 100, mode = "regression") %>%
  set_engine("randomForest") 

housing_model_randomForest
Random Forest Model Specification (regression)

Main Arguments:
  trees = 100

Computational engine: randomForest 
housing_fit_randomForest <- housing_model_randomForest %>%
  fit(price ~ ., data = housing_training_juiced)

housing_fit_randomForest
parsnip model object


Call:
 randomForest(x = as.data.frame(x), y = y, ntree = ~100) 
               Type of random forest: regression
                     Number of trees: 100
No. of variables tried at each split: 2

          Mean of squared residuals: 6238597435
                    % Var explained: 64.71
# Define and fit a random forest regression model using the ranger engine/package
housing_model_ranger <- rand_forest(trees = 100, mode = "regression") %>%
  set_engine("ranger") 

housing_model_ranger
Random Forest Model Specification (regression)

Main Arguments:
  trees = 100

Computational engine: ranger 
housing_fit_ranger <- housing_model_ranger %>%
  fit(price ~ ., data = housing_training_juiced)

housing_fit_ranger
parsnip model object

Ranger result

Call:
 ranger::ranger(formula = formula, data = data, num.trees = ~100,      num.threads = 1, verbose = FALSE, seed = sample.int(10^5,          1)) 

Type:                             Regression 
Number of trees:                  100 
Sample size:                      699 
Number of independent variables:  7 
Mtry:                             2 
Target node size:                 5 
Variable importance mode:         none 
Splitrule:                        variance 
OOB prediction error (MSE):       6103318793 
R squared (OOB):                  0.655258

Once we have the model fits, we can use the predict function to generate our predictions for our testing dataset. The predict function always produces a dataframe with the same number of rows as observations. Because of this, bind_cols can be used to bind the predictions to the original dataframe

# Generate predictions for our testing using the ranger model
predict(housing_fit_ranger, housing_testing_baked)

# Add these ranger predictions to the testing dataset
housing_fit_ranger %>%
  predict(housing_testing_baked) %>%
  bind_cols(housing_testing_baked)

# Save this combined dataframe for later
housing_ranger_predict <- housing_fit_ranger %>%
  predict(housing_testing_baked) %>%
  bind_cols(housing_testing_baked) %>%
  # Add a column for model name
  mutate(model_name = "ranger")

# Let's do the same thing for the linerar regression model
housing_lm_predict <- housing_fit_lm %>%
  predict(housing_testing_baked) %>%
  bind_cols(housing_testing_baked) %>%
  mutate(model_name = "lm")

# Let's do the same thing for the randomForest model
housing_randomForest_predict <- housing_fit_randomForest %>%
  predict(housing_testing_baked) %>%
  bind_cols(housing_testing_baked) %>%
  mutate(model_name = "randomForest")

# Let's combine all of these datasets so we can look at them side-by-side
housing_all_predict <- bind_rows(housing_lm_predict,
                              housing_ranger_predict,
                              housing_randomForest_predict)

Let’s just look and see how our predictions line up with the observed values in our testing dataset.

housing_all_predict %>%
  ggplot(aes(x = price,y=.pred,color=model_name)) +
  geom_point() +
  geom_smooth(method = "lm") +
  labs(x = "Observed price",
       y = "Predicted price",
       title = "Predictions vs observed values for 3 model types\nA simple linear regression is overlaid") +
  coord_equal()

Model performance assessment

Using our predictions from the parsnip package, we can use the yardstick package to generate model performance metrics. This can be done using the metrics function, which generates a default metric set (for a regression model, these are root mean squared error or rsme, r-squared or rsq, and mean absolute error or mae; for a classification model, these are accuracy and Kappa or kap). You can also define a custom set of metrics using metric_set, and there are also individual functions for all metric types.

housing_ranger_predict %>%
  # Here we use the metric functions and must define the truth value and the estimated prediction
  metrics(truth = price, estimate = .pred)

When the predictions are in a dataframe, we can group by model type and calculat metrics by group

housing_all_predict%>%
  group_by(model_name) %>%
  metrics(truth = price, estimate = .pred)

housing_all_predict%>%
  group_by(model_name) %>%
  metrics(truth = price, estimate = .pred)%>%
  ggplot(aes(x = model_name, y = .estimate)) +
  geom_bar(stat="identity") +
  facet_wrap(.~.metric,scales="free") +
  labs(x = "Model name",
       y = "Model performance metric estimate",
       title = "Model performance metrics for 3 model types")

We can also do what we just did in a much more tidy fashion, while also keeping the model specifications, model fits, model predictions, and model metrics all in a single dataframe. This ensures that things stay together, and makes it very easy to extract summary statistics or plots. purrr::map and list columns makes this all possible. We could apply this same approach to build and test many models for cross-validation, for hyperparameter tuning, etc.

# Define a tibble using model names and their associated specifications
set.seed(101)
all_models <- 
  tibble(model_name = "lm",
         model = list(housing_model_lm)) %>%
  add_row(model_name = "ranger",
          model = list(housing_model_ranger)) %>%
  add_row(model_name = "randomForest",
          model = list(housing_model_randomForest))

all_models

all_model_results <- all_models %>%
  # Add a column for model fits
  mutate(model_fit = purrr::map(model,
                                ~fit(.x, price ~ ., data = housing_training_juiced)),
         # Add a column for predictions
         model_predictions = purrr::map(model_fit,
                                        ~.x %>% 
                                          predict(housing_testing_baked) %>%
                                          bind_cols(housing_testing_baked)),
         # Add a column for model metrics
         model_metrics = purrr::map(model_predictions,
                                    ~metrics(.x, truth = price, estimate = .pred)))

all_model_results

# This plot is the same as the one we made above
all_model_results %>%
  unnest(model_metrics) %>%
  ggplot(aes(x = model_name, y = .estimate)) +
  geom_bar(stat="identity") +
  facet_wrap(.~.metric,scales="free") +
  labs(x = "Model name",
       y = "Model performance metric estimate",
       title = "Model performance metrics for 3 model types")

Classification example using GermanCredit

Let’s also go through a classification example using the GermanCredit dataset from caret. Now we will try to predict credit rating (good or bad) using a number of predictors.

Load packages

data(GermanCredit)

GermanCredit

Data pre-processing

# Split the credit dataset, using 75% of the data for training and 25% for testing, stratified by credit class
# This maintains the ratio of Good and Bad credit classes in both the training and testing datasets
credit_split <- GermanCredit %>%
  as_tibble() %>%
  # Convert most columns to factors since they are binaries
  mutate_at(vars(-Duration,-Amount,-InstallmentRatePercentage,-ResidenceDuration,-Age,-NumberExistingCredits,-NumberPeopleMaintenance),
            as.factor) %>%
  initial_split(prop = 0.75, strata = "Class")

# The training function can be used to extract the training data from the rsplit object
credit_training <- credit_split %>%
  training()

# The testing function can be used to extract the testing data from the rsplit object
credit_testing <- credit_split %>%
  testing()

credit_recipe <- credit_training %>%
  recipe(Class ~.) %>%
  # Remove all near-zero variance predictors, such as factors with only one level
  step_nzv(all_predictors()) %>%
  # step_corr removes highly correlated variables
  step_corr(all_numeric()) %>%
  # step_center normalizes data to have a mean of 0
  step_center(all_numeric()) %>%
  # step_scale normalizes data to have a standard deviation of 0
  step_scale(all_numeric())%>%
  # Make all factors dummy columns
  step_dummy(all_nominal(),-all_outcomes()) 

credit_recipe_prepped <- credit_recipe %>%
  # prep trains the recipe using the training dataset
  prep()

# Use use the juice function to apply the prepped recipe to the training dataset
credit_training_juiced <- juice(credit_recipe_prepped)

# We use the bake function to apply the prepped recipe to the testing dataset
credit_testing_baked <- credit_recipe_prepped %>%
  bake(credit_testing)

Model training

# Define and fit a logistic regression model
credit_model_lr <- logistic_reg() %>%
  set_engine("glm") 

# Define and fit a random forest regression model using the randomForest engine/package
credit_model_randomForest <-  rand_forest(trees = 100, mode = "classification") %>%
  set_engine("randomForest") 

# Define and fit a random forest regression model using the ranger engine/package
credit_model_ranger <- rand_forest(trees = 100, mode = "classification") %>%
  set_engine("ranger") 


# Define a tibble using model names and their associated specifications
all_models_credit <- 
  tibble(model_name = "ranger",
          model = list(credit_model_ranger)) %>%
  add_row(model_name = "randomForest",
          model = list(credit_model_randomForest))

all_model_results_credit <- all_models_credit %>%
  # Add a column for model fits
  mutate(model_fit = purrr::map(model,
                                ~fit(.x, Class ~ ., data = credit_training_juiced)),
         # Add a column for class predictions
         model_predictions_class = purrr::map(model_fit,
                                        ~.x %>% 
                                          predict(credit_testing_baked, type = "class") %>%
                                          bind_cols(credit_testing_baked)),
         # Add a column for model metrics from class predictions
         model_metrics = purrr::map(model_predictions_class,
                                    ~metrics(.x, truth = Class, estimate = .pred_class)),
         # Add a column for probability predictions
         model_predictions_prob = purrr::map(model_fit,
                                              ~.x %>% 
                                                predict(credit_testing_baked, type="prob") %>%
                                                bind_cols(credit_testing_baked)),
         # Add ROC curves from probability predictions
         roc_curves = purrr::map(model_predictions_prob,
                                ~roc_curve(.x, Class, .pred_Good)))

Model performance assessment

# Let's plot the performance metrics by model type
all_model_results_credit %>%
  unnest(model_metrics) %>%
  ggplot(aes(x = model_name, y = .estimate)) +
  geom_bar(stat="identity") +
  facet_wrap(.~.metric,scales="free") +
  labs(x = "Model name",
       y = "Model performance metric estimate",
       title = "Model performance metrics for 2 model types")


# Let's also plot ROC curves by model type
all_model_results_credit %>%
  unnest(roc_curves) %>%
  ggplot(aes(x = 1-specificity,y=sensitivity,color=model_name)) +
  geom_line() +
  geom_abline(slope=1) +
  labs(title = "Receiver Operating Characteristic (ROC) curves for 2 model types",
    x = "False positive rate\n[1 - specificity = 1 - TN/(TN + FP)]",
    y = "True positive rate\n[recall = sensitivity = TP / (TP + FN)]")

LS0tCnRpdGxlOiAiVGlkeW1vZGVscyBsaXZlIHNlc3Npb24gLSBFY29EYXRhU2NpZW5jZSIKYXV0aG9yOiAiR2F2aW4gTWNEb25hbGQgLSBFbnZpcm9ubWVudGFsIE1hcmtldHMgU29sdXRpb25zIExhYiAoZW1MYWIpIgpkYXRlOiAiMTEvMTkvMjAxOSIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKIyBJbnRyb2R1Y3Rpb24gCgpUaGlzIGxpdmUgc2Vzc2lvbiBoYXMgYmVlbiBhZGFwdGVkIGZyb20gZnJvbSBFZGdhciBSdWl6J3MgW0EgR2VudGxlIEludHJvZHVjdGlvbiB0byB0aWR5bW9kZWxzXShodHRwczovL3J2aWV3cy5yc3R1ZGlvLmNvbS8yMDE5LzA2LzE5L2EtZ2VudGxlLWludHJvLXRvLXRpZHltb2RlbHMvKS4gV2Ugd2lsbCBydW4gdGhyb3VnaCBhbiBlbmQtdG8tZW5kIG1vZGVsaW5nIGV4YW1wbGVzLiBUaGUgZmlyc3Qgd2lsbCBiZSBhIHJlZ3Jlc3Npb24gZXhhbXBsZSB1c2luZyB0aGUgYFNhY3JhbWVudG9gIGhvdXNpbmcgcHJpY2VzIGRhdGFzZXQsIGFuZCB0aGUgc2Vjb25kIHdpbGwgYmUgYSBjbGFzc2lmaWNhdGlvbiBleGFtcGxlIHVzaW5nIHRoZSBgR2VybWFuQ3JlZGl0YCBjcmVkaXQgc2NvcmUgZGF0YXNldC4gCgpJIHdpbGwgYnJpZWZseSB0b3VjaCBvbiBpbXBvcnRhbnQgc3RlcHMgb2YgdGhlIHByZWRpY3RpdmUgbW9kZWxpbmcgcHJvY2VzcywgYnV0IHRoaXMgaXMgKm5vdCogbWVhbiB0byBiZSBhIGNvbXByZWhlbnNpdmUgaW5zdHJ1Y3Rpb24gb24gcHJlZGljdGl2ZSBtb2RlbGluZyAoKmEuay5hLiogbWFjaGluZSBsZWFybmluZykuIFRoaXMgdHV0b3JpYWwgaXMgcmF0aGVyIGp1c3QgbWVhbnQgdG8gZ2l2ZSBhIGZsYXZvciBvZiB3aGF0J3MgcG9zc2libGUgdXNpbmcgdGhlIG5ldyBgdGlkeW1vZGVsc2AgdW5pdmVyc2Ugb2YgcGFja2FnZXMuIEZvciBhIG11Y2ggbW9yZSBkZXRhaWxlZCBvdmVydmlldywgc2VlIE1heCBLdWhuJ3MgZmFudGFzdGljIFtBcHBsaWVkIFByZWRpY3RpdmUgTW9kZWxpbmddKGh0dHA6Ly9hcHBsaWVkcHJlZGljdGl2ZW1vZGVsaW5nLmNvbSkuCgojIEluc3RhbGwgbmVjZXNzYXJ5IHBhY2FrZ2VzCgpgYGB7cn0KIyBwYWNtYW4gd2lsbCBoZWxwIHVzIGluc3RhbGwgYW55IG5lY2Vzc2FyeSBwYWNrYWdlcwppZiAoIXJlcXVpcmUoInBhY21hbiIpKSBpbnN0YWxsLnBhY2thZ2VzKCJwYWNtYW4iKQojIHBhY21hbjo6cF9sb2FkIGNoZWNrcyB0byBzZWUgaWYgdGhpcyBwYWNrYWdlcyBhcmUgaW5zdGFsbGVkLCBhbmQgaW5zdGFsbHMgdGhlbSBpZiBub3QKcGFjbWFuOjpwX2xvYWQodGlkeW1vZGVscywgcmFuZ2VyLCByYW5kb21Gb3Jlc3QsIGNhcmV0KQpgYGAKCgojIExvYWQgcGFja2FnZXMKCkxvYWQgdGhlIHRpZHltb2RlbHMgbGlicmFyeS4gVGhpcyBsb2FkcyBhIGNvbGxlY3Rpb24gb2YgYm90aCB0aWR5bW9kZWxzIHBhY2thZ2VzLCBhbmQgc2VsZWN0IHRpZHl2ZXJzZSBwYWNrYWdlcyBsaWtlIGRwbHlyLCBwdXJyciwgZ2dwbG90Mi4gV2Ugd2lsbCBhbHNvIGxvYWQgY2FyZXQgc2luY2UgaXQgaGFzIHNvbWUgdmVyeSBbbmljZSBkYXRhc2V0c10oaHR0cHM6Ly90b3BlcG8uZ2l0aHViLmlvL2NhcmV0L2RhdGEtc2V0cy5odG1sKSBmb3IgcmVncmVzc2lvbiBhbmQgY2xhc3NpZmljYXRpb24gZXhlcmNpc2VzLgoKYGBge3J9CmxpYnJhcnkodGlkeW1vZGVscykKbGlicmFyeShjYXJldCkKIyBTZXQgcmFuZG9tIG51bWJlciBzZWVkIHRvIGdldCBjb25zaXN0ZW50IHJlc3VsdHMKc2V0LnNlZWQoMTAxKQpgYGAKCiMgUmVncmVzc2lvbiBleGFtcGxlIHVzaW5nIGBTYWNyYW1lbnRvYAoKIyMgTG9hZCBkYXRhc2V0CgpgYGB7cn0KZGF0YShTYWNyYW1lbnRvKQoKU2FjcmFtZW50bwpgYGAKCiMjIERhdGEgc2FtcGxpbmcgCgpUaGUgZmlyc3Qgc3RlcCBpbiB0aGUgbW9kZWxpbmcgcHJvY2VzcyBpcyB0byBzcGxpdCB5b3VyIGRhdGEgaW50byBzZXBhcmF0ZSB0cmFpbmluZyBhbmQgdGVzdGluZyBkYXRhc2V0cy4gVGhlIG1vZGVsIHdpbGwgYmUgdHJhaW5lZCB1c2luZyB0aGUgdHJhaW5pbmcgZGF0YXNldCwgYW5kIHRoZSB0ZXN0aW5nIGRhdGFzZXQgd2lsbCBub3QgYmUgdXNlZCB1bnRpbCB5b3UgYXJlIHJlYWR5IHRvIGFzc2VzcyBtb2RlbCBwZXJmb3JtYW5jZS4gVGhlIGByc2FtcGxlOjppbml0aWFsX3NwbGl0YCBmdW5jdGlvbiBoZWxwcyB3aXRoIHRoaXMgaW5pdGlhbCBzcGxpdHRpbmcgb2YgdGhlIGRhdGEuIFRoZSBgcnNhbXBsZWAgaW5jbHVkZXMgbWFueSBvdGhlciBoZWxwZnVsIGZ1bmN0aW9ucyBmb3Igc3BsaXR0aW5nIHRoZSBkYXRhIGZvciBjcm9zcy12YWxpZGF0aW9uLCBib290c3RyYXBwaW5nLCBldGMuCgpgYGB7cn0KIyBTcGxpdCB0aGUgZGF0YXNldCwgdXNpbmcgNzUlIG9mIHRoZSBkYXRhIGZvciB0cmFpbmluZyBhbmQgMjUlIGZvciB0ZXN0aW5nCmhvdXNpbmdfc3BsaXQgPC0gU2FjcmFtZW50byAlPiUKICBhc190aWJibGUoKSAlPiUKICBkcGx5cjo6c2VsZWN0KHByaWNlLHR5cGUsc3FmdCxiZWRzLGJhdGhzLGxhdGl0dWRlLGxvbmdpdHVkZSkgJT4lCiAgaW5pdGlhbF9zcGxpdChwcm9wID0gMC43NSkKCiMgVGhpcyByc3BsaXQgb2JqZWN0IHRlbGxzIHlvdSBob3cgbWFueSBvYnNlcnZhdGlvbnMgYXJlIHVzZWQgZm9yIHRyYWluaW5nLCBob3cgbWFueSBmb3IgdGVzdGluZywgYW5kIGhvdyBtYW55IHRvdGFsCmhvdXNpbmdfc3BsaXQKCiMgVGhlIHRyYWluaW5nIGZ1bmN0aW9uIGNhbiBiZSB1c2VkIHRvIGV4dHJhY3QgdGhlIHRyYWluaW5nIGRhdGEgZnJvbSB0aGUgcnNwbGl0IG9iamVjdApob3VzaW5nX3RyYWluaW5nIDwtIGhvdXNpbmdfc3BsaXQgJT4lCiAgdHJhaW5pbmcoKQoKaG91c2luZ190cmFpbmluZwoKIyBUaGUgdGVzdGluZyBmdW5jdGlvbiBjYW4gYmUgdXNlZCB0byBleHRyYWN0IHRoZSB0ZXN0aW5nIGRhdGEgZnJvbSB0aGUgcnNwbGl0IG9iamVjdApob3VzaW5nX3Rlc3RpbmcgPC0gaG91c2luZ19zcGxpdCAlPiUKICB0ZXN0aW5nKCkKCmhvdXNpbmdfdGVzdGluZwpgYGAKCiMjIERhdGEgcHJlLXByb2Nlc3NpbmcgCgpBZnRlciBzcGxpdHRpbmcgdGhlIGRhdGEsIHdlIHdpbGwgZG8gc29tZSBkYXRhIHByb2Nlc3NpbmcuIFRvIGRvIHRoaXMsIHdlIHdpbGwgdXNlIHRoZSBgcmVjaXBlYCBwYWNrYWdlLiBBIHJlY2lwZSBpcyBhIGJsdWVwcmludCBmb3IgaG93IGRhdGEgd2lsbCBiZSBwcm9jZXNzZWQuIEJ5IGNyZWF0aW5nIGEgYmx1ZXByaW50LCByYXRoZXIgdGhhbiBwcm9jZXNzaW5nIGRhdGEgZGlyZWN0bHksIHdlIGNhbiBhcHBseSB0aGUgc2FtZSBibHVlcHJpbnQgdG8gdHJhaW5pbmcgYW5kIHRlc3RpbmcgZGF0YXNldHMuIEltcG9ydGFudGx5LCB0aGUgcmVjaXBlIGlzIGRlZmluZWQgdXNpbmcgb25seSBkYXRhIGZyb20gdGhlIHRyYWluaW5nIGRhdGFzZXQsIHdoaWNoIHdpbGwgYWxsb3cgdXMgdG8gc2VlIGhvdyB3ZWxsIHRoZSBtb2RlbCBwZXJmb3JtcyB1c2luZyB0aGUgdGVzdGluZyBkYXRhc2V0LiBSZWNpcGUgc3RlcHMgY2FuIGJlIGRlZmluZWQgdXNpbmcgcGlwZXMgd2l0aCBhIG51bWJlciBvZiBzZXF1ZW50aWFsIHN0ZXBzIC0gdGhlcmUgYXJlIG1hbnkgbWFueSBvcHRpb25zIGZvciByZWNpcGUgc3RlcHMuCgpgYGB7cn0KCmhvdXNpbmdfcmVjaXBlIDwtIGhvdXNpbmdfdHJhaW5pbmcgJT4lCiAgIyBTcGVjaWZ5IHJlZ3Jlc3Npb24gbW9kZWwgZm9ybXVsYQogIHJlY2lwZShwcmljZSB+LikgICU+JQogICMgc3RlcF9jb3JyIHJlbW92ZXMgaGlnaGx5IGNvcnJlbGF0ZWQgdmFyaWFibGVzCiAgc3RlcF9jb3JyKGFsbF9udW1lcmljKCkpICU+JQogICMgc3RlcF9jZW50ZXIgbm9ybWFsaXplcyBkYXRhIHRvIGhhdmUgYSBtZWFuIG9mIDAKICBzdGVwX2NlbnRlcihhbGxfbnVtZXJpYygpLCAtYWxsX291dGNvbWVzKCkpICU+JQogICMgc3RlcF9zY2FsZSBub3JtYWxpemVzIGRhdGEgdG8gaGF2ZSBhIHN0YW5kYXJkIGRldmlhdGlvbiBvZiAwCiAgc3RlcF9zY2FsZShhbGxfbnVtZXJpYygpLCAtYWxsX291dGNvbWVzKCkpICU+JQogICMgQ3JlYXRlIGR1bW15IHZhcmlhYmxlIGNvbHVtbnMgZm9yIGFsbCBmYWN0b3IgY29sdW1ucwogIHN0ZXBfZHVtbXkoYWxsX3ByZWRpY3RvcnMoKSwtYWxsX251bWVyaWMoKSkKCmhvdXNpbmdfcmVjaXBlCgpob3VzaW5nX3JlY2lwZV9wcmVwcGVkIDwtIGhvdXNpbmdfcmVjaXBlICU+JQogICMgcHJlcCB0cmFpbnMgdGhlIHJlY2lwZSB1c2luZyB0aGUgdHJhaW5pbmcgZGF0YXNldAogIHByZXAoKQoKaG91c2luZ19yZWNpcGVfcHJlcHBlZAoKIyBVc2UgdXNlIHRoZSBqdWljZSBmdW5jdGlvbiB0byBhcHBseSB0aGUgcHJlcHBlZCByZWNpcGUgdG8gdGhlIHRyYWluaW5nIGRhdGFzZXQKaG91c2luZ190cmFpbmluZ19qdWljZWQgPC0ganVpY2UoaG91c2luZ19yZWNpcGVfcHJlcHBlZCkKCmhvdXNpbmdfdHJhaW5pbmdfanVpY2VkCgojIFdlIHVzZSB0aGUgYmFrZSBmdW5jdGlvbiB0byBhcHBseSB0aGUgcHJlcHBlZCByZWNpcGUgdG8gdGhlIHRlc3RpbmcgZGF0YXNldApob3VzaW5nX3Rlc3RpbmdfYmFrZWQgPC0gaG91c2luZ19yZWNpcGVfcHJlcHBlZCAlPiUKICBiYWtlKGhvdXNpbmdfdGVzdGluZykgCgpob3VzaW5nX3Rlc3RpbmdfYmFrZWQKYGBgCgojIyBNb2RlbCB0cmFpbmluZwoKTmV4dCwgd2Ugd2lsbCB1c2UgdGhlIGBwYXJzbmlwYCBwYWNrYWdlIHRvIGRlZmluZSBhIG51bWJlciBvZiBtb2RlbHMuIEdlbmVyYWxseSwgd2UgdXNlIHBhcnNuaXAgdG8gZGVmaW5lIDMgdGhpbmdzIGFib3V0IG91ciBtb2RlbDoKCjEuIFRoZSB0eXBlIG9mIG1vZGVsIChlLmcuLCBsaW5lYXIgcmVncmVzc2lvbiBvciByYW5kb20gZm9yZXN0KSAgCjIuIHRoZSBtb2RlIG9mIHRoZSBtb2RlbCAoZS5nLiwgcmVncmVzc2lvbiBvciBjbGFzc2lmaWNhdGlvbikgIAozLiBUaGUgZW5naW5lIGZvciB0aGUgbW9kZWwgKGUuZy4sIGByYW5nZXJgIG9yIGByYW5kb21Gb3Jlc3RgKSAKCkFmdGVyIHdlJ3ZlIGRlZmluZWQgdGhlIG1vZGVsIGluIHRoaXMgd2F5LCB3ZSBjYW4gdXNlIHRoZSBgZml0YCBmdW5jdGlvbiB0byBmaXQgdGhlIG1vZGVsLiAgCgpgYGB7cn0KIyBEZWZpbmUgYW5kIGZpdCBhIGxpbmVhciByZWdyZXNzaW9uIG1vZGVsCmhvdXNpbmdfbW9kZWxfbG0gPC0gbGluZWFyX3JlZygpICU+JQogIHNldF9lbmdpbmUoImxtIikgCgpob3VzaW5nX21vZGVsX2xtCgpob3VzaW5nX2ZpdF9sbSA8LSBob3VzaW5nX21vZGVsX2xtICU+JQogIGZpdChwcmljZSB+IC4sIGRhdGEgPSBob3VzaW5nX3RyYWluaW5nX2p1aWNlZCkKCmhvdXNpbmdfZml0X2xtCgojIERlZmluZSBhbmQgZml0IGEgcmFuZG9tIGZvcmVzdCByZWdyZXNzaW9uIG1vZGVsIHVzaW5nIHRoZSByYW5kb21Gb3Jlc3QgZW5naW5lL3BhY2thZ2UKaG91c2luZ19tb2RlbF9yYW5kb21Gb3Jlc3QgPC0gIHJhbmRfZm9yZXN0KHRyZWVzID0gMTAwLCBtb2RlID0gInJlZ3Jlc3Npb24iKSAlPiUKICBzZXRfZW5naW5lKCJyYW5kb21Gb3Jlc3QiKSAKCmhvdXNpbmdfbW9kZWxfcmFuZG9tRm9yZXN0Cgpob3VzaW5nX2ZpdF9yYW5kb21Gb3Jlc3QgPC0gaG91c2luZ19tb2RlbF9yYW5kb21Gb3Jlc3QgJT4lCiAgZml0KHByaWNlIH4gLiwgZGF0YSA9IGhvdXNpbmdfdHJhaW5pbmdfanVpY2VkKQoKaG91c2luZ19maXRfcmFuZG9tRm9yZXN0CgojIERlZmluZSBhbmQgZml0IGEgcmFuZG9tIGZvcmVzdCByZWdyZXNzaW9uIG1vZGVsIHVzaW5nIHRoZSByYW5nZXIgZW5naW5lL3BhY2thZ2UKaG91c2luZ19tb2RlbF9yYW5nZXIgPC0gcmFuZF9mb3Jlc3QodHJlZXMgPSAxMDAsIG1vZGUgPSAicmVncmVzc2lvbiIpICU+JQogIHNldF9lbmdpbmUoInJhbmdlciIpIAoKaG91c2luZ19tb2RlbF9yYW5nZXIKCmhvdXNpbmdfZml0X3JhbmdlciA8LSBob3VzaW5nX21vZGVsX3JhbmdlciAlPiUKICBmaXQocHJpY2UgfiAuLCBkYXRhID0gaG91c2luZ190cmFpbmluZ19qdWljZWQpCgpob3VzaW5nX2ZpdF9yYW5nZXIKYGBgCgpPbmNlIHdlIGhhdmUgdGhlIG1vZGVsIGZpdHMsIHdlIGNhbiB1c2UgdGhlIGBwcmVkaWN0YCBmdW5jdGlvbiB0byBnZW5lcmF0ZSBvdXIgcHJlZGljdGlvbnMgZm9yIG91ciB0ZXN0aW5nIGRhdGFzZXQuIFRoZSBwcmVkaWN0IGZ1bmN0aW9uIGFsd2F5cyBwcm9kdWNlcyBhIGRhdGFmcmFtZSB3aXRoIHRoZSBzYW1lIG51bWJlciBvZiByb3dzIGFzIG9ic2VydmF0aW9ucy4gQmVjYXVzZSBvZiB0aGlzLCBgYmluZF9jb2xzYCBjYW4gYmUgdXNlZCB0byBiaW5kIHRoZSBwcmVkaWN0aW9ucyB0byB0aGUgb3JpZ2luYWwgZGF0YWZyYW1lCgpgYGB7cn0KIyBHZW5lcmF0ZSBwcmVkaWN0aW9ucyBmb3Igb3VyIHRlc3RpbmcgdXNpbmcgdGhlIHJhbmdlciBtb2RlbApwcmVkaWN0KGhvdXNpbmdfZml0X3JhbmdlciwgaG91c2luZ190ZXN0aW5nX2Jha2VkKQoKIyBBZGQgdGhlc2UgcmFuZ2VyIHByZWRpY3Rpb25zIHRvIHRoZSB0ZXN0aW5nIGRhdGFzZXQKaG91c2luZ19maXRfcmFuZ2VyICU+JQogIHByZWRpY3QoaG91c2luZ190ZXN0aW5nX2Jha2VkKSAlPiUKICBiaW5kX2NvbHMoaG91c2luZ190ZXN0aW5nX2Jha2VkKQoKIyBTYXZlIHRoaXMgY29tYmluZWQgZGF0YWZyYW1lIGZvciBsYXRlcgpob3VzaW5nX3Jhbmdlcl9wcmVkaWN0IDwtIGhvdXNpbmdfZml0X3JhbmdlciAlPiUKICBwcmVkaWN0KGhvdXNpbmdfdGVzdGluZ19iYWtlZCkgJT4lCiAgYmluZF9jb2xzKGhvdXNpbmdfdGVzdGluZ19iYWtlZCkgJT4lCiAgIyBBZGQgYSBjb2x1bW4gZm9yIG1vZGVsIG5hbWUKICBtdXRhdGUobW9kZWxfbmFtZSA9ICJyYW5nZXIiKQoKIyBMZXQncyBkbyB0aGUgc2FtZSB0aGluZyBmb3IgdGhlIGxpbmVyYXIgcmVncmVzc2lvbiBtb2RlbApob3VzaW5nX2xtX3ByZWRpY3QgPC0gaG91c2luZ19maXRfbG0gJT4lCiAgcHJlZGljdChob3VzaW5nX3Rlc3RpbmdfYmFrZWQpICU+JQogIGJpbmRfY29scyhob3VzaW5nX3Rlc3RpbmdfYmFrZWQpICU+JQogIG11dGF0ZShtb2RlbF9uYW1lID0gImxtIikKCiMgTGV0J3MgZG8gdGhlIHNhbWUgdGhpbmcgZm9yIHRoZSByYW5kb21Gb3Jlc3QgbW9kZWwKaG91c2luZ19yYW5kb21Gb3Jlc3RfcHJlZGljdCA8LSBob3VzaW5nX2ZpdF9yYW5kb21Gb3Jlc3QgJT4lCiAgcHJlZGljdChob3VzaW5nX3Rlc3RpbmdfYmFrZWQpICU+JQogIGJpbmRfY29scyhob3VzaW5nX3Rlc3RpbmdfYmFrZWQpICU+JQogIG11dGF0ZShtb2RlbF9uYW1lID0gInJhbmRvbUZvcmVzdCIpCgojIExldCdzIGNvbWJpbmUgYWxsIG9mIHRoZXNlIGRhdGFzZXRzIHNvIHdlIGNhbiBsb29rIGF0IHRoZW0gc2lkZS1ieS1zaWRlCmhvdXNpbmdfYWxsX3ByZWRpY3QgPC0gYmluZF9yb3dzKGhvdXNpbmdfbG1fcHJlZGljdCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaG91c2luZ19yYW5nZXJfcHJlZGljdCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaG91c2luZ19yYW5kb21Gb3Jlc3RfcHJlZGljdCkKYGBgCgpMZXQncyBqdXN0IGxvb2sgYW5kIHNlZSBob3cgb3VyIHByZWRpY3Rpb25zIGxpbmUgdXAgd2l0aCB0aGUgb2JzZXJ2ZWQgdmFsdWVzIGluIG91ciB0ZXN0aW5nIGRhdGFzZXQuCgpgYGB7cn0KaG91c2luZ19hbGxfcHJlZGljdCAlPiUKICBnZ3Bsb3QoYWVzKHggPSBwcmljZSx5PS5wcmVkLGNvbG9yPW1vZGVsX25hbWUpKSArCiAgZ2VvbV9wb2ludCgpICsKICBnZW9tX3Ntb290aChtZXRob2QgPSAibG0iKSArCiAgbGFicyh4ID0gIk9ic2VydmVkIHByaWNlIiwKICAgICAgIHkgPSAiUHJlZGljdGVkIHByaWNlIiwKICAgICAgIHRpdGxlID0gIlByZWRpY3Rpb25zIHZzIG9ic2VydmVkIHZhbHVlcyBmb3IgMyBtb2RlbCB0eXBlc1xuQSBzaW1wbGUgbGluZWFyIHJlZ3Jlc3Npb24gaXMgb3ZlcmxhaWQiKSArCiAgY29vcmRfZXF1YWwoKQpgYGAKCgojIyBNb2RlbCBwZXJmb3JtYW5jZSBhc3Nlc3NtZW50CgpVc2luZyBvdXIgcHJlZGljdGlvbnMgZnJvbSB0aGUgYHBhcnNuaXBgIHBhY2thZ2UsIHdlIGNhbiB1c2UgdGhlIGB5YXJkc3RpY2tgIHBhY2thZ2UgdG8gZ2VuZXJhdGUgbW9kZWwgcGVyZm9ybWFuY2UgbWV0cmljcy4gVGhpcyBjYW4gYmUgZG9uZSB1c2luZyB0aGUgYG1ldHJpY3NgIGZ1bmN0aW9uLCB3aGljaCBnZW5lcmF0ZXMgYSBkZWZhdWx0IG1ldHJpYyBzZXQgKGZvciBhIHJlZ3Jlc3Npb24gbW9kZWwsIHRoZXNlIGFyZSByb290IG1lYW4gc3F1YXJlZCBlcnJvciBvciBgcnNtZWAsIHItc3F1YXJlZCBvciBgcnNxYCwgYW5kIG1lYW4gYWJzb2x1dGUgZXJyb3Igb3IgYG1hZWA7IGZvciBhIGNsYXNzaWZpY2F0aW9uIG1vZGVsLCB0aGVzZSBhcmUgYGFjY3VyYWN5YCBhbmQgS2FwcGEgb3IgYGthcGApLiBZb3UgY2FuIGFsc28gZGVmaW5lIGEgY3VzdG9tIHNldCBvZiBtZXRyaWNzIHVzaW5nIGBtZXRyaWNfc2V0YCwgYW5kIHRoZXJlIGFyZSBhbHNvIGluZGl2aWR1YWwgZnVuY3Rpb25zIGZvciBhbGwgbWV0cmljIHR5cGVzLgoKYGBge3J9CmhvdXNpbmdfcmFuZ2VyX3ByZWRpY3QgJT4lCiAgIyBIZXJlIHdlIHVzZSB0aGUgbWV0cmljIGZ1bmN0aW9ucyBhbmQgbXVzdCBkZWZpbmUgdGhlIHRydXRoIHZhbHVlIGFuZCB0aGUgZXN0aW1hdGVkIHByZWRpY3Rpb24KICBtZXRyaWNzKHRydXRoID0gcHJpY2UsIGVzdGltYXRlID0gLnByZWQpCmBgYAoKV2hlbiB0aGUgcHJlZGljdGlvbnMgYXJlIGluIGEgZGF0YWZyYW1lLCB3ZSBjYW4gZ3JvdXAgYnkgbW9kZWwgdHlwZSBhbmQgY2FsY3VsYXQgbWV0cmljcyBieSBncm91cAoKYGBge3J9CmhvdXNpbmdfYWxsX3ByZWRpY3QlPiUKICBncm91cF9ieShtb2RlbF9uYW1lKSAlPiUKICBtZXRyaWNzKHRydXRoID0gcHJpY2UsIGVzdGltYXRlID0gLnByZWQpCgpob3VzaW5nX2FsbF9wcmVkaWN0JT4lCiAgZ3JvdXBfYnkobW9kZWxfbmFtZSkgJT4lCiAgbWV0cmljcyh0cnV0aCA9IHByaWNlLCBlc3RpbWF0ZSA9IC5wcmVkKSU+JQogIGdncGxvdChhZXMoeCA9IG1vZGVsX25hbWUsIHkgPSAuZXN0aW1hdGUpKSArCiAgZ2VvbV9iYXIoc3RhdD0iaWRlbnRpdHkiKSArCiAgZmFjZXRfd3JhcCgufi5tZXRyaWMsc2NhbGVzPSJmcmVlIikgKwogIGxhYnMoeCA9ICJNb2RlbCBuYW1lIiwKICAgICAgIHkgPSAiTW9kZWwgcGVyZm9ybWFuY2UgbWV0cmljIGVzdGltYXRlIiwKICAgICAgIHRpdGxlID0gIk1vZGVsIHBlcmZvcm1hbmNlIG1ldHJpY3MgZm9yIDMgbW9kZWwgdHlwZXMiKQpgYGAKCldlIGNhbiBhbHNvIGRvIHdoYXQgd2UganVzdCBkaWQgaW4gYSBtdWNoIG1vcmUgdGlkeSBmYXNoaW9uLCB3aGlsZSBhbHNvIGtlZXBpbmcgdGhlIG1vZGVsIHNwZWNpZmljYXRpb25zLCBtb2RlbCBmaXRzLCBtb2RlbCBwcmVkaWN0aW9ucywgYW5kIG1vZGVsIG1ldHJpY3MgYWxsIGluIGEgc2luZ2xlIGRhdGFmcmFtZS4gVGhpcyBlbnN1cmVzIHRoYXQgdGhpbmdzIHN0YXkgdG9nZXRoZXIsIGFuZCBtYWtlcyBpdCB2ZXJ5IGVhc3kgdG8gZXh0cmFjdCBzdW1tYXJ5IHN0YXRpc3RpY3Mgb3IgcGxvdHMuIGBwdXJycjo6bWFwYCBhbmQgbGlzdCBjb2x1bW5zIG1ha2VzIHRoaXMgYWxsIHBvc3NpYmxlLiBXZSBjb3VsZCBhcHBseSB0aGlzIHNhbWUgYXBwcm9hY2ggdG8gYnVpbGQgYW5kIHRlc3QgbWFueSBtb2RlbHMgZm9yIGNyb3NzLXZhbGlkYXRpb24sIGZvciBoeXBlcnBhcmFtZXRlciB0dW5pbmcsIGV0Yy4KCmBgYHtyfQojIERlZmluZSBhIHRpYmJsZSB1c2luZyBtb2RlbCBuYW1lcyBhbmQgdGhlaXIgYXNzb2NpYXRlZCBzcGVjaWZpY2F0aW9ucwphbGxfbW9kZWxzIDwtIAogIHRpYmJsZShtb2RlbF9uYW1lID0gImxtIiwKICAgICAgICAgbW9kZWwgPSBsaXN0KGhvdXNpbmdfbW9kZWxfbG0pKSAlPiUKICBhZGRfcm93KG1vZGVsX25hbWUgPSAicmFuZ2VyIiwKICAgICAgICAgIG1vZGVsID0gbGlzdChob3VzaW5nX21vZGVsX3JhbmdlcikpICU+JQogIGFkZF9yb3cobW9kZWxfbmFtZSA9ICJyYW5kb21Gb3Jlc3QiLAogICAgICAgICAgbW9kZWwgPSBsaXN0KGhvdXNpbmdfbW9kZWxfcmFuZG9tRm9yZXN0KSkKCmFsbF9tb2RlbHMKCmFsbF9tb2RlbF9yZXN1bHRzIDwtIGFsbF9tb2RlbHMgJT4lCiAgIyBBZGQgYSBjb2x1bW4gZm9yIG1vZGVsIGZpdHMKICBtdXRhdGUobW9kZWxfZml0ID0gcHVycnI6Om1hcChtb2RlbCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB+Zml0KC54LCBwcmljZSB+IC4sIGRhdGEgPSBob3VzaW5nX3RyYWluaW5nX2p1aWNlZCkpLAogICAgICAgICAjIEFkZCBhIGNvbHVtbiBmb3IgcHJlZGljdGlvbnMKICAgICAgICAgbW9kZWxfcHJlZGljdGlvbnMgPSBwdXJycjo6bWFwKG1vZGVsX2ZpdCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIH4ueCAlPiUgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByZWRpY3QoaG91c2luZ190ZXN0aW5nX2Jha2VkKSAlPiUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYmluZF9jb2xzKGhvdXNpbmdfdGVzdGluZ19iYWtlZCkpLAogICAgICAgICAjIEFkZCBhIGNvbHVtbiBmb3IgbW9kZWwgbWV0cmljcwogICAgICAgICBtb2RlbF9tZXRyaWNzID0gcHVycnI6Om1hcChtb2RlbF9wcmVkaWN0aW9ucywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfm1ldHJpY3MoLngsIHRydXRoID0gcHJpY2UsIGVzdGltYXRlID0gLnByZWQpKSkKCmFsbF9tb2RlbF9yZXN1bHRzCgojIFRoaXMgcGxvdCBpcyB0aGUgc2FtZSBhcyB0aGUgb25lIHdlIG1hZGUgYWJvdmUKYWxsX21vZGVsX3Jlc3VsdHMgJT4lCiAgdW5uZXN0KG1vZGVsX21ldHJpY3MpICU+JQogIGdncGxvdChhZXMoeCA9IG1vZGVsX25hbWUsIHkgPSAuZXN0aW1hdGUpKSArCiAgZ2VvbV9iYXIoc3RhdD0iaWRlbnRpdHkiKSArCiAgZmFjZXRfd3JhcCgufi5tZXRyaWMsc2NhbGVzPSJmcmVlIikgKwogIGxhYnMoeCA9ICJNb2RlbCBuYW1lIiwKICAgICAgIHkgPSAiTW9kZWwgcGVyZm9ybWFuY2UgbWV0cmljIGVzdGltYXRlIiwKICAgICAgIHRpdGxlID0gIk1vZGVsIHBlcmZvcm1hbmNlIG1ldHJpY3MgZm9yIDMgbW9kZWwgdHlwZXMiKQpgYGAKCiMgQ2xhc3NpZmljYXRpb24gZXhhbXBsZSB1c2luZyBgR2VybWFuQ3JlZGl0YAoKTGV0J3MgYWxzbyBnbyB0aHJvdWdoIGEgY2xhc3NpZmljYXRpb24gZXhhbXBsZSB1c2luZyB0aGUgYEdlcm1hbkNyZWRpdGAgZGF0YXNldCBmcm9tIGBjYXJldGAuIE5vdyB3ZSB3aWxsIHRyeSB0byBwcmVkaWN0IGNyZWRpdCByYXRpbmcgKGdvb2Qgb3IgYmFkKSB1c2luZyBhIG51bWJlciBvZiBwcmVkaWN0b3JzLgoKIyMgTG9hZCBwYWNrYWdlcwpgYGB7cn0KZGF0YShHZXJtYW5DcmVkaXQpCgpHZXJtYW5DcmVkaXQKYGBgCgojIyBEYXRhIHByZS1wcm9jZXNzaW5nIApgYGAge3J9CiMgU3BsaXQgdGhlIGNyZWRpdCBkYXRhc2V0LCB1c2luZyA3NSUgb2YgdGhlIGRhdGEgZm9yIHRyYWluaW5nIGFuZCAyNSUgZm9yIHRlc3RpbmcsIHN0cmF0aWZpZWQgYnkgY3JlZGl0IGNsYXNzCiMgVGhpcyBtYWludGFpbnMgdGhlIHJhdGlvIG9mIEdvb2QgYW5kIEJhZCBjcmVkaXQgY2xhc3NlcyBpbiBib3RoIHRoZSB0cmFpbmluZyBhbmQgdGVzdGluZyBkYXRhc2V0cwpjcmVkaXRfc3BsaXQgPC0gR2VybWFuQ3JlZGl0ICU+JQogIGFzX3RpYmJsZSgpICU+JQogICMgQ29udmVydCBtb3N0IGNvbHVtbnMgdG8gZmFjdG9ycyBzaW5jZSB0aGV5IGFyZSBiaW5hcmllcwogIG11dGF0ZV9hdCh2YXJzKC1EdXJhdGlvbiwtQW1vdW50LC1JbnN0YWxsbWVudFJhdGVQZXJjZW50YWdlLC1SZXNpZGVuY2VEdXJhdGlvbiwtQWdlLC1OdW1iZXJFeGlzdGluZ0NyZWRpdHMsLU51bWJlclBlb3BsZU1haW50ZW5hbmNlKSwKICAgICAgICAgICAgYXMuZmFjdG9yKSAlPiUKICBpbml0aWFsX3NwbGl0KHByb3AgPSAwLjc1LCBzdHJhdGEgPSAiQ2xhc3MiKQoKIyBUaGUgdHJhaW5pbmcgZnVuY3Rpb24gY2FuIGJlIHVzZWQgdG8gZXh0cmFjdCB0aGUgdHJhaW5pbmcgZGF0YSBmcm9tIHRoZSByc3BsaXQgb2JqZWN0CmNyZWRpdF90cmFpbmluZyA8LSBjcmVkaXRfc3BsaXQgJT4lCiAgdHJhaW5pbmcoKQoKIyBUaGUgdGVzdGluZyBmdW5jdGlvbiBjYW4gYmUgdXNlZCB0byBleHRyYWN0IHRoZSB0ZXN0aW5nIGRhdGEgZnJvbSB0aGUgcnNwbGl0IG9iamVjdApjcmVkaXRfdGVzdGluZyA8LSBjcmVkaXRfc3BsaXQgJT4lCiAgdGVzdGluZygpCgpjcmVkaXRfcmVjaXBlIDwtIGNyZWRpdF90cmFpbmluZyAlPiUKICByZWNpcGUoQ2xhc3Mgfi4pICU+JQogICMgUmVtb3ZlIGFsbCBuZWFyLXplcm8gdmFyaWFuY2UgcHJlZGljdG9ycywgc3VjaCBhcyBmYWN0b3JzIHdpdGggb25seSBvbmUgbGV2ZWwKICBzdGVwX256dihhbGxfcHJlZGljdG9ycygpKSAlPiUKICAjIHN0ZXBfY29yciByZW1vdmVzIGhpZ2hseSBjb3JyZWxhdGVkIHZhcmlhYmxlcwogIHN0ZXBfY29ycihhbGxfbnVtZXJpYygpKSAlPiUKICAjIHN0ZXBfY2VudGVyIG5vcm1hbGl6ZXMgZGF0YSB0byBoYXZlIGEgbWVhbiBvZiAwCiAgc3RlcF9jZW50ZXIoYWxsX251bWVyaWMoKSkgJT4lCiAgIyBzdGVwX3NjYWxlIG5vcm1hbGl6ZXMgZGF0YSB0byBoYXZlIGEgc3RhbmRhcmQgZGV2aWF0aW9uIG9mIDAKICBzdGVwX3NjYWxlKGFsbF9udW1lcmljKCkpJT4lCiAgIyBNYWtlIGFsbCBmYWN0b3JzIGR1bW15IGNvbHVtbnMKICBzdGVwX2R1bW15KGFsbF9ub21pbmFsKCksLWFsbF9vdXRjb21lcygpKSAKCmNyZWRpdF9yZWNpcGVfcHJlcHBlZCA8LSBjcmVkaXRfcmVjaXBlICU+JQogICMgcHJlcCB0cmFpbnMgdGhlIHJlY2lwZSB1c2luZyB0aGUgdHJhaW5pbmcgZGF0YXNldAogIHByZXAoKQoKIyBVc2UgdXNlIHRoZSBqdWljZSBmdW5jdGlvbiB0byBhcHBseSB0aGUgcHJlcHBlZCByZWNpcGUgdG8gdGhlIHRyYWluaW5nIGRhdGFzZXQKY3JlZGl0X3RyYWluaW5nX2p1aWNlZCA8LSBqdWljZShjcmVkaXRfcmVjaXBlX3ByZXBwZWQpCgojIFdlIHVzZSB0aGUgYmFrZSBmdW5jdGlvbiB0byBhcHBseSB0aGUgcHJlcHBlZCByZWNpcGUgdG8gdGhlIHRlc3RpbmcgZGF0YXNldApjcmVkaXRfdGVzdGluZ19iYWtlZCA8LSBjcmVkaXRfcmVjaXBlX3ByZXBwZWQgJT4lCiAgYmFrZShjcmVkaXRfdGVzdGluZykgCmBgYAoKIyMgTW9kZWwgdHJhaW5pbmcgIAoKYGBge3J9CiMgRGVmaW5lIGFuZCBmaXQgYSBsb2dpc3RpYyByZWdyZXNzaW9uIG1vZGVsCmNyZWRpdF9tb2RlbF9sciA8LSBsb2dpc3RpY19yZWcoKSAlPiUKICBzZXRfZW5naW5lKCJnbG0iKSAKCiMgRGVmaW5lIGFuZCBmaXQgYSByYW5kb20gZm9yZXN0IHJlZ3Jlc3Npb24gbW9kZWwgdXNpbmcgdGhlIHJhbmRvbUZvcmVzdCBlbmdpbmUvcGFja2FnZQpjcmVkaXRfbW9kZWxfcmFuZG9tRm9yZXN0IDwtICByYW5kX2ZvcmVzdCh0cmVlcyA9IDEwMCwgbW9kZSA9ICJjbGFzc2lmaWNhdGlvbiIpICU+JQogIHNldF9lbmdpbmUoInJhbmRvbUZvcmVzdCIpIAoKIyBEZWZpbmUgYW5kIGZpdCBhIHJhbmRvbSBmb3Jlc3QgcmVncmVzc2lvbiBtb2RlbCB1c2luZyB0aGUgcmFuZ2VyIGVuZ2luZS9wYWNrYWdlCmNyZWRpdF9tb2RlbF9yYW5nZXIgPC0gcmFuZF9mb3Jlc3QodHJlZXMgPSAxMDAsIG1vZGUgPSAiY2xhc3NpZmljYXRpb24iKSAlPiUKICBzZXRfZW5naW5lKCJyYW5nZXIiKSAKCgojIERlZmluZSBhIHRpYmJsZSB1c2luZyBtb2RlbCBuYW1lcyBhbmQgdGhlaXIgYXNzb2NpYXRlZCBzcGVjaWZpY2F0aW9ucwphbGxfbW9kZWxzX2NyZWRpdCA8LSAKICB0aWJibGUobW9kZWxfbmFtZSA9ICJyYW5nZXIiLAogICAgICAgICAgbW9kZWwgPSBsaXN0KGNyZWRpdF9tb2RlbF9yYW5nZXIpKSAlPiUKICBhZGRfcm93KG1vZGVsX25hbWUgPSAicmFuZG9tRm9yZXN0IiwKICAgICAgICAgIG1vZGVsID0gbGlzdChjcmVkaXRfbW9kZWxfcmFuZG9tRm9yZXN0KSkKCmFsbF9tb2RlbF9yZXN1bHRzX2NyZWRpdCA8LSBhbGxfbW9kZWxzX2NyZWRpdCAlPiUKICAjIEFkZCBhIGNvbHVtbiBmb3IgbW9kZWwgZml0cwogIG11dGF0ZShtb2RlbF9maXQgPSBwdXJycjo6bWFwKG1vZGVsLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIH5maXQoLngsIENsYXNzIH4gLiwgZGF0YSA9IGNyZWRpdF90cmFpbmluZ19qdWljZWQpKSwKICAgICAgICAgIyBBZGQgYSBjb2x1bW4gZm9yIGNsYXNzIHByZWRpY3Rpb25zCiAgICAgICAgIG1vZGVsX3ByZWRpY3Rpb25zX2NsYXNzID0gcHVycnI6Om1hcChtb2RlbF9maXQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB+LnggJT4lIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBwcmVkaWN0KGNyZWRpdF90ZXN0aW5nX2Jha2VkLCB0eXBlID0gImNsYXNzIikgJT4lCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGJpbmRfY29scyhjcmVkaXRfdGVzdGluZ19iYWtlZCkpLAogICAgICAgICAjIEFkZCBhIGNvbHVtbiBmb3IgbW9kZWwgbWV0cmljcyBmcm9tIGNsYXNzIHByZWRpY3Rpb25zCiAgICAgICAgIG1vZGVsX21ldHJpY3MgPSBwdXJycjo6bWFwKG1vZGVsX3ByZWRpY3Rpb25zX2NsYXNzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB+bWV0cmljcygueCwgdHJ1dGggPSBDbGFzcywgZXN0aW1hdGUgPSAucHJlZF9jbGFzcykpLAogICAgICAgICAjIEFkZCBhIGNvbHVtbiBmb3IgcHJvYmFiaWxpdHkgcHJlZGljdGlvbnMKICAgICAgICAgbW9kZWxfcHJlZGljdGlvbnNfcHJvYiA9IHB1cnJyOjptYXAobW9kZWxfZml0LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfi54ICU+JSAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJlZGljdChjcmVkaXRfdGVzdGluZ19iYWtlZCwgdHlwZT0icHJvYiIpICU+JQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBiaW5kX2NvbHMoY3JlZGl0X3Rlc3RpbmdfYmFrZWQpKSwKICAgICAgICAgIyBBZGQgUk9DIGN1cnZlcyBmcm9tIHByb2JhYmlsaXR5IHByZWRpY3Rpb25zCiAgICAgICAgIHJvY19jdXJ2ZXMgPSBwdXJycjo6bWFwKG1vZGVsX3ByZWRpY3Rpb25zX3Byb2IsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfnJvY19jdXJ2ZSgueCwgQ2xhc3MsIC5wcmVkX0dvb2QpKSkKYGBgCgojIyBNb2RlbCBwZXJmb3JtYW5jZSBhc3Nlc3NtZW50CgpgYGB7cn0KIyBMZXQncyBwbG90IHRoZSBwZXJmb3JtYW5jZSBtZXRyaWNzIGJ5IG1vZGVsIHR5cGUKYWxsX21vZGVsX3Jlc3VsdHNfY3JlZGl0ICU+JQogIHVubmVzdChtb2RlbF9tZXRyaWNzKSAlPiUKICBnZ3Bsb3QoYWVzKHggPSBtb2RlbF9uYW1lLCB5ID0gLmVzdGltYXRlKSkgKwogIGdlb21fYmFyKHN0YXQ9ImlkZW50aXR5IikgKwogIGZhY2V0X3dyYXAoLn4ubWV0cmljLHNjYWxlcz0iZnJlZSIpICsKICBsYWJzKHggPSAiTW9kZWwgbmFtZSIsCiAgICAgICB5ID0gIk1vZGVsIHBlcmZvcm1hbmNlIG1ldHJpYyBlc3RpbWF0ZSIsCiAgICAgICB0aXRsZSA9ICJNb2RlbCBwZXJmb3JtYW5jZSBtZXRyaWNzIGZvciAyIG1vZGVsIHR5cGVzIikKCiMgTGV0J3MgYWxzbyBwbG90IFJPQyBjdXJ2ZXMgYnkgbW9kZWwgdHlwZQphbGxfbW9kZWxfcmVzdWx0c19jcmVkaXQgJT4lCiAgdW5uZXN0KHJvY19jdXJ2ZXMpICU+JQogIGdncGxvdChhZXMoeCA9IDEtc3BlY2lmaWNpdHkseT1zZW5zaXRpdml0eSxjb2xvcj1tb2RlbF9uYW1lKSkgKwogIGdlb21fbGluZSgpICsKICBnZW9tX2FibGluZShzbG9wZT0xKSArCiAgbGFicyh0aXRsZSA9ICJSZWNlaXZlciBPcGVyYXRpbmcgQ2hhcmFjdGVyaXN0aWMgKFJPQykgY3VydmVzIGZvciAyIG1vZGVsIHR5cGVzIiwKICAgIHggPSAiRmFsc2UgcG9zaXRpdmUgcmF0ZVxuWzEgLSBzcGVjaWZpY2l0eSA9IDEgLSBUTi8oVE4gKyBGUCldIiwKICAgIHkgPSAiVHJ1ZSBwb3NpdGl2ZSByYXRlXG5bcmVjYWxsID0gc2Vuc2l0aXZpdHkgPSBUUCAvIChUUCArIEZOKV0iKQoKYGBgCgo=